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Schools topic area and meets WWC group design standards with res- 
ervations. This study included 2,446 students in grades 9-12 in one school district in the United States. No studies 
meet WWC group design standards without reservations. 


According to the WWC review, the extent of evidence for Green Dot Public Schools on the educational outcomes 
of students was small for four outcome domains— mathematics achievement, student progression, school atten- 
dance, and English language arts achievement. No studies meet WWC group design standards in the six other 
domains, so this intervention report does not report on the effectiveness of Green Dot Public Schools for those 
domains.° (See the Effectiveness Summary on p. 5 for more details of effectiveness by domain.) 


Effectiveness 


Green Dot Public Schools had potentially positive effects on mathematics achievement, student progression, 
school attendance, and English language arts achievement for high school students. 
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Table 1. Summary of findings* 


Improvement index (percentile points) 


Number of Number of Extent of 
Outcome domain Rating of effectiveness Average Range studies students® evidence 
Mathematics Potentially positive effects +13 na 1 2,103 Small 
achievement 
Student progression Potentially positive effects +10 na 1 1,024 Small 
School attendance Potentially positive effects +5 na 1 1,578 Small 
English language Potentially positive effects +4 na 1 1,865 Small 


arts achievement 


na = not applicable 
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Intervention Information 
Background 


Green Dot Public Schools was founded in 1999 and continues to operate both start-up charter schools and turn- 
around charter schools. Address: 1149 S. Hill Street, Suite 600, Los Angeles, CA 90015. Web: greendot.org. Tele- 
phone: (823) 565-1600. 


Intervention details 


The Green Dot Public Schools model creates small community high schools and emphasizes a set of six principles. 
The first is to recruit, train, and support teachers who can accelerate student learning and close the achievement 
gap. The second is to offer autonomous school leadership to principals and assistant principals who are respon- 
sible for identifying high-quality teachers and providing them with personalized coaching and professional support. 
The third is to promote a college-going culture and offer a college preparatory curriculum. The fourth is to provide 
a range of comprehensive supports, including health and wellness services, gang intervention and mentorship pro- 
grams, and college and career services. The fifth is to engage parents through adult education programming and 
self-advocacy training. Lastly, the Green Dot Public Schools model aims to be replicable in similar schools serving 
low-income students with unionized teacher workforces and similar funding sources. 


Some schools in the Green Dot Public Schools network were chartered in existing public schools which were per- 
forming below district or community expectations. The study summarized in this report examined a group of Green 
Dot Public Schools formed by dividing an existing public high school into a set of smaller schools. 


Cost 


Green Dot Public Schools receive public funding to cover day-to-day operations. Transformation operations that 
transition traditional schools into the control of Green Dot Public Schools might have additional costs covered 
largely by private foundations. For example, Green Dot Public Schools spent approximately $15 million over the 
course of a 4-year turnaround of a single high school campus in Los Angeles (Dillon, 2010), mainly from private 
sources. 
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Research Summary 


The WWC identified one eligible study that investigated the effects of Table 2. Scope of reviewed research 
Green Dot Public Schools on the academic outcomes of students. One 


Grade 9-|2 
additional study was identified but did not meet eligibility criteria (see aT hod Io eanal 
the Glossary of Terms in this document for a definition of this term and eee sews 

Intervention type Program 


other commonly used research terms) for review in this topic area. Cita- 
tions for these two studies are in the References section, which begins 
on p. 8.° 


The WWC reviewed one eligible study against group design standards. This study uses a quasi-experimental 
design that meets WWC group design standards with reservations. This report summarizes this one study. 


Summary of studies meeting WWC group design standards without reservations 
No studies of Green Dot Public Schools met WWC group design standards without reservations. 


Summary of studies meeting WWC group design standards with reservations 


Herman et al. (2012) conducted a quasi-experimental study that examined the effects of Green Dot Public Schools 
on students in 11 high schools in the Los Angeles Unified School District (LAUSD), comparing students from eight 
Green Dot Public Schools to students from three comparsion schools. This study meets WWC standards with 
reservations because it uses a quasi-experimental design in which the analytic intervention and comparison groups 
are shown to be equivalent. The study included three cohorts of students entering ninth grade and measured 
outcomes for up to 4 years. The authors compared students starting ninth grade in Green Dot Public Schools with 
similar students who started high school in the three comparison schools, matched using propensity scores based 
on gender, ethnicity, parents’ education, poverty status, language classification, eighth-grade California Standards 
Test (CST) scores and math subject taken, school attendance, and whether students had attended middle school in 
the same set of feeder middle schools as students in Green Dot Public Schools. The authors compared outcomes 
between the two groups within each cohort and across each year of high school. The analytic sample included 386 
students who started ninth grade in 2007-08 (Cohort 1), 1,024 students who started ninth grade in 2008-09 (Cohort 
2), and 1,036 students who started ninth grade in 2009-10 (Cohort 3).’ 
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Effectiveness Summary 


The WWC review of Green Dot Public Schools for the Charter Schools topic area includes outcomes in 10 
domains.® The one study of Green Dot Public Schools that met WWC group design standards reported findings 

in four domains: mathematics achievement, student progression, school attendance, and English language arts 
achievement. The following findings present the authors’ estimates and WWC-calculated estimates of the size and 
statistical significance of the effects of Green Dot Public Schools on students. For outcomes measured at multiple 
points in time for the entire sample, the primary findings that the WWC considered for the effectiveness rating are 
those measured furthest from the initial exposure to the intervention to reflect the maximum exposure of students 
to the program. Because we identify the primary finding separately for each sample and outcome measure from 
among those findings that meet WWC group design standards, the primary findings within an outcome domain are 
sometimes drawn from different time periods. In cases where outcomes were measured for different portions of 
the overall sample at different times, each outcome for each sample was considered a primary finding. Additional 
comparisons are available as supplemental findings in Appendix D. The supplemental findings do not factor into 
the intervention’s rating of effectiveness. For a more detailed description of the rating of effectiveness and extent of 
evidence criteria, see the WWC Rating Criteria on p. 24. 


Summary of effectiveness for the mathematics achievement domain 


Table 3. Rating of effectiveness and extent of evidence for the mathematics achievement domain 


Rating of effectiveness Criteria met 


Potentially positive effects In the one study that reported findings, the estimated impact of the intervention on outcomes in the mathematics 
Evidence of a positive effect with achievement domain was positive and statistically significant. 
no overriding contrary evidence. 


Extent of evidence Criteria met 


Small One study that included 2,103 students in 11 schools reported evidence of effectiveness in the mathematics 
achievement domain. 


One study that met WWC group design standards with reservations reported findings in the mathematics achieve- 
ment domain. 


Herman et al. (2012) found a statistically significant positive effect of attending Green Dot Public Schools on Cali- 
fornia Standards Test (CST) scores in Algebra | and Geometry in the first year of high school for students across 
all three cohorts. The authors also reported statistically significant positive effects of attending Green Dot Public 
Schools on the math scale score of the California High School Exit Exam (CAHSEE) in the second year of high 
school for students who started ninth grade in 2007-08 (Cohort 1) and 2008-09 (Cohort 2). The study also found 
statistically significant positive effects of attending Green Dot Public Schools on CST scores in Algebra Il and 
Geometry in the second year, and in Algebra II and Summative Math in the third year for students in Cohorts 1 
and 2.° The WWC applied a correction for multiple comparisons, but this did not affect the statistical significance 
among any of the contrasts. The WWC characterizes this finding as a potentially positive effect. 


Thus, for the mathematics achievement domain, one study showed statistically significant effects. This results in a 
rating of potentially positive effects, that is based on a small extent of evidence. 
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Summary of effectiveness for the student progression domain 


Table 4. Rating of effectiveness and extent of evidence for the student progression domain 


Rating of effectiveness Criteria met 


Potentially positive effects In the one study that reported findings, the estimated impact of the intervention on outcomes in the student 
Evidence of a positive effect with progression domain was statistically significant. 
no overriding contrary evidence. 


Extent of evidence Criteria met 
Small One study that included 1,024 students in 11 schools reported evidence of effectiveness in the student progres- 
sion domain. 


One study that met WWC group design standards with reservations reported findings in the student progression 
domain. 


Herman et al. (2012) reported statistically significant positive effects of attending Green Dot Public Schools on 
student progression. Students who started ninth grade in Green Dot Public Schools in 2008-09 (Cohort 2) were 
significantly more likely than comparison students to graduate from high school and to graduate with A-G require- 
ments, the set of high school courses necessary for admissions to California public universities. The WWC applied 
a correction for multiple comparisons, but this did not affect the statistical significance for either contrast. The 
WWC characterizes this finding as a potentially positive effect. 


Thus, for the student progression domain, one study showed statistically significant effects. This results in a rating 
of potentially positive effects, that is based on a small extent of evidence. 


Summary of effectiveness for the school attendance domain 


Table 5. Rating of effectiveness and extent of evidence for the school attendance domain 


Rating of effectiveness Criteria met 


Potentially positive effects In the one study that reported findings, the estimated impact of the intervention on outcomes in the school 
Evidence of a positive effect with attendance domain was statistically significant. 
no overriding contrary evidence. 


Extent of evidence Criteria met 
Small One study that included 1,578 students in 11 schools reported evidence of effectiveness in the school attendance 
domain. 


One study that met WWC group design standards with reservations reported findings in the school attendance 
domain. 


Herman et al. (2012) reported statistically significant positive effects of attending Green Dot Public Schools on 
school attendance. Students who started at Green Dot Public Schools in 2007-08 (Cohort 1) and 2008-09 (Cohort 
2) had significantly higher rates of attendance during the fourth year of high school than comparison students. The 
authors report, and the WWC confirmed, no statistically significant or substantively important effects of Green Dot 
Public Schools on school attendance in Year 1 among students who started ninth grade in 2009-10 (Cohort 3). The 
WWC characterizes this study as finding a potentially positive effect. 


Thus, for the school attendance domain, one study showed statistically significant effects. This results in a rating of 
potentially positive effects, that is based on a small extent of evidence. 
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Summary of effectiveness for the English language arts achievement domain 
Table 6. Rating of effectiveness and extent of evidence for the English language arts achievement domain 


Rating of effectiveness Criteria met 


Potentially positive effects In the one study that reported findings, the estimated impact of the intervention on outcomes in the English 
Evidence of a positive effect with language arts achievement domain was statistically significant. 
no overriding contrary evidence. 


Extent of evidence Criteria met 


Small One study that included 1,865 students in 11 schools reported evidence of effectiveness in the English language 
arts achievement domain. 


One study that met WWC group design standards with reservations reported findings in the English language arts 
domain. 


Herman et al. (2012) reported statistically significant positive effects of attending a Green Dot Public School on CST 
scores in English language arts during the first year of high school for students who started ninth grade in Green 
Dot Public Schools in 2009-10 (Cohort 3). The authors reported, and the WWC confirmed, no statistically signifi- 
cant or substantively important effects on CST scores in ELA during the third year of high school for students who 
started ninth grade in Green Dot Public Schools in 2007-08 (Cohort 1) or 2008-09 (Cohort 2). The study also found 
no statistically significant or substantively important effects of attending Green Dot Public Schools on the English 
Language Arts scale score of the CAHSEE in the second year of high school for Cohorts 1 and 2. The WWC applied 
a correction for multiple comparisons, but this did not affect the statistical significance among any of the contrasts. 
The WWC characterizes this study finding as a potentially positive effect. 


Thus, for the English language arts achievement domain, one study showed statistically significant effects. This 
results in a rating of potentially positive effects, that is based on a small extent of evidence. 
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Studies that do not meet WWC group design standards 
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Study that is ineligible for review using the Charter Schools Evidence Review Protocol 
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Appendix A: Research details for Herman et al. (2012) 


Herman, J. L., Wang, J., Rickles, J., Hsu, V., Monroe, S., Leon, S., ... CRESST. (2012). Evaluation of 
Green Dot’s Locke Transformation Project: Findings for Cohort 1 and 2 students (CRESST Report 
815). Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Test- 
ing (CRESST). Retrieved from https://eric.ed.gov/?id=ED531993 


Additional sources: 


Herman, J., Rickles, J., Hansen, M., Thomas, L., Gualpa, A., Wang, J., & CRESST. (2011). Evaluation 
of Green Dot’s Locke Transformation Project: Findings for the 2007-08, 2008-09, and 2009-10 
School Years (CRESST Report 799). National Center for Research on Evaluation, Standards, 
and Student Testing (CRESST). Retrieved from hitps://eric.ed.gov/?id=ED522836 

Rickles, J., Wang, J., Herman, J., & CRESST. (2013). Evaluation of Green Dot’s Locke Transforma- 
tion Project: Supplemental report on Cohort 2 student outcomes (CRESST Report 825). Los 
Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing 
(CRESST). Retrieved from https://eric.ed.gov/?id=ED540596 


Table A. Summary of findings Meets WWC Group Design Standards With Reservations 
Study findings 
Average improvement index 
Outcome domain Sample size (percentile points) Statistically significant 
Mathematics achievement 2,103 students +13 Yes 
Student progression 1,024 students +10 Yes 
School attendance 1,578 students +5 Yes 
English language arts 1,865 students +4 Yes 


achievement 


Setting The study took place in 11 urban high schools in the Los Angeles Unified School District 
(LAUSD) in Los Angeles, California. Eight schools were Green Dot Public Schools, and three 
were traditional LAUSD high schools. 


Study sample __ The study included three cohorts based on year of entry into high school. Cohort 1 entered 
ninth grade during the 2007-08 school year, Cohort 2 started during the 2008-09 school year, 
and Cohort 3 started during the 2009-10 school year. Beginning in the fall of 2007, Green 
Dot started converting a single, existing LAUSD high school into Green Dot Public Schools. 
This transition started with two off-site small schools in the first year. By fall 2008, the total 
neighborhood catchment and full student community that was previously associated with the 
original LAUSD school became students in eight Green Dot Public Schools. The study used 
a quasi-experimental design to match Green Dot Public Schools students to similar students 
in three traditional high schools who did not have the opportunity to attend the eight Green 
Dot Public Schools. The authors first identified a set of six LAUSD middle schools where 
the majority of students in Green Dot Public Schools had attended eighth grade and the 
corresponding three traditional high schools where most students in those middle schools 
started ninth grade. The students in these traditional high schools formed the pool of potential 
matches for students in Green Dot Public Schools. Each student enrolled in Green Dot Public 
Schools was matched exactly to another student from the same cohort enrolled in one of 
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Intervention 
group 


Comparison 
group 


Outcomes and 
measurement 


the traditional high schools based on whether they had the same gender, ethnicity, parents’ 
education, poverty status, language classification, math subject exam taken on the eighth 
grade California Standards Test (CST), and had attended middle school in the same set of 
feeder middle schools as students in Green Dot Public Schools. When more than one student 
was identified as a possible match, the authors selected one to be included in the comparison 
group using nearest-neighbor propensity score matching based on eighth-grade CST scale 
scores in math and ELA and on eighth-grade attendance rates. 


Across the three cohorts, 71%-88% of students were eligible for free or reduced-price lunch, 
31%-46% were classified as English language learners, 6%-9% of students had a disability, 
74%-79% were Latino, 21%-26% were African American, and 51%-52% were female. 


Students in the intervention condition attended one of eight high schools in the Green Dot 
Public Schools network. The schools in the intervention condition were previously a single tra- 
ditional LAUSD school that Green Dot Public Schools converted into multiple charter schools 
over a 2-year period. The school was divided into two schools for the first year (the 2007-08 
school year), and into eight schools for the second year (the 2008-09 school year). 


Students in the comparison condition attended one of three traditional LAUSD schools. 


The study measured outcomes at the end of each of 4 years of high school for Cohorts 1 and 
2, and the end of the first year of high school for Cohort 3. 


Mathematics achievement was measured using state standardized tests. The CST was admin- 
istered across three math subjects: Algebra | in Year 1, Geometry in Years 1 and 2, Algebra II 
in Years 2 and 3, and Summative Math in Year 3. Each student took only one of these math 
subject tests per year, and different groups of students took each subject test every year (for 
example, some students in a cohort took the Geometry test in Year 1, and others took the test 
in Year 2). The study also included results from the CAHSEE in math, administered to students 
for the first time at the end of Year 2. Progression outcomes were the graduation rate and 
graduation rate after completing a sequence of college preparatory courses within 4 years of 
entering high school. School attendance was measured in Year 1 for Cohort 3 and in Year 4 for 
Cohorts 1 and 2. English language arts achievement was measured using state standardized 
tests. The CST in English language arts was administered in Year 1 for Cohort 3 and in Year 

3 for Cohorts 1 and 2. The study also included results from the CAHSEE in English language 
arts, administered to students for the first time at the end of Year 2. For a more detailed 
description of these outcome measures, see Appendix B. 


Supplemental findings include measurement of outcomes in different years for each cohort, 
and outcomes based on different samples of students. These additional outcomes are 
reported as supplemental findings in Appendix D, but do not factor into the intervention’s rat- 
ing of effectiveness. 


While the findings from Cohort 2 that contribute to the effectiveness rating are based ona 
sample with 4 years of outcomes available (results presented in Rickles et al., 2013), findings 
from Cohort 2 based on an alternate sample with only 3 years of outcomes (results presented 
in Herman et al., 2012) are reported as supplemental findings. 


Green Dot Public Schools January 2018 Page 10 


WWC Intervention Report 


Results for Algebra II CST scores in Year 2 for Cohort 1 and Geometry CST scores in Year 1 for 
both Cohort 2 and the alternate Cohort 2 did not meet WWC group design standards. These 
samples were not shown to be equivalent at baseline across the intervention and comparison 
groups, and therefore are not included in this review. 


The study collected outcomes on course-taking and completion, which were not eligible 
outcomes for review under the Charter Schools review protocol (version 3.0). The authors also 
measured semester-to-semester persistence, which was not eligible for review under the pro- 
tocol because it measures school mobility rather than student progression. In addition to the 
CAHSEE outcomes described above, the authors measured passing in two or more attempts 
in both the third and fourth years of high school. The WWC determined these outcomes were 
not eligible for review because they were conditional on failing the exam on the first attempt. 
Students who passed the exam on the first attempt or after multiple attempts are included in 
the passing on any attempt version of the outcome. 


Support for § The Green Dot Public Schools model emphasizes partnerships with parents, the school 
implementation district, and the community. Teachers and principals are provided with recommended best 
practices to achieve the goals of the organization. 
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Appendix B: Outcome measures for each domain 


Mathematics achievement 


California Standards Test (CST): Score on the CST exam in Algebra | administered in the first year of high school (as cited in Rickles et al., 2013). 

Algebra | (Year 1) The Algebra | test is one of a set of criterion-referenced standardized tests in the CST administered to California 
students in grades 2-11 from 2002-14 and aligned to state-adopted content standards. 

CST: Geometry (Year 7 or 2) Score on the CST exam in Geometry, which can be administered in the first or second year of high school (as 
cited in Rickles et al., 2013). 

CST: Algebra II (Year 2 or 3) Score on the CST exam in Algebra Il administered in the second or third year of high school (as cited in Rickles 
et al., 2013). 

CST: Summative Math (Year 3) Score on the CST exam in Summative Math administered in the third year of high school (as cited in Rickles et 


al., 2013). The exam is administered to students who have already completed Algebra II and covers content from 
Algebra | and II, Geometry, and Probability and Statistics. 


California High School Exit Exam Score on the CAHSEE in math administered in the second year of high school (as cited in Rickles et al., 2013). 
(CAHSEE): Math scale score (Year 2) Beginning with the class of 2006, students in California public schools were required to pass the CAHSEE 
to demonstrate competency in grade-level skills in reading, writing, and mathematics to earn a high school 
diploma. 


CAHSEE: Ever passed math (Year 3 or 4) | Passing the CAHSEE is a binary measure of whether a student ever passed the math section of the exam, which 
is measured at the end of both the third and fourth years of high school (as cited in Rickles et al., 2013). The 
CAHSEE is first administered in the second year of high school, but students can take the test up to three times 
per school year up to a total of six times. This outcome is only reported as a supplemental finding. 


CAHSEE: Passed math on first attempt Passing the CAHSEE on the first attempt is a binary measure of whether a student passed the math section 
(Year 2) of the exam on the first attempt during the second year of high school (as cited in Rickles et al., 2013). This 
outcome is only reported as a supplemental finding. 


Student progression 


Graduated (since entry in Year 1 Graduating from high school is a binary measure of whether a student is observed graduating from any school in 

or since Year 4 enrollment) the district by the end of the fourth year of high school (as cited in Rickles et al., 2013) for those enrolled in their 
school at either the beginning of Year 1 or Year 4. All Year 1 students are entering ninth-grade students, and all 
Year 4 students have stayed enrolled in their same schools from Year 1 to Year 4. 


Graduated with A-G (since entry in Graduating from high school with A-G is a binary measure of whether a student graduated and completed a 

Year 1 or since Year 4 enrollment) college preparatory sequence of courses by the end of the fourth year of high school (as cited in Rickles et al., 
2013) for those enrolled in their school at either the beginning of Year 1 or Year 4. All Year 1 students are 
entering ninth-grade students, and all Year 4 students have stayed enrolled in their same schools from Year 1 to 
Year 4. 


School attendance 


School attendance (Year 1, 2, 3, or 4) School attendance is the average of the number of days a student attended divided by the total number of 
school days for students enrolled in school at the beginning of each year. For each year, the outcome only 
included students who were still enrolled in the school up to and through that year (as cited in Herman et al., 


2012). 
English language arts 
CST: English language arts Score on the CST exam in the subject of English language arts administered in the first, second, and third years 
(Year 1, 2, or 3) of high school (as cited in Rickles et al., 2013). 
CAHSEE: English language arts Scale score on the CAHSEE in the subject of English language arts administered in the second year of high 
scale score (Year 2) school (as cited in Rickles et al., 2013). 
CAHSEE: Ever passed English Passing the CAHSEE is a binary measure of whether a student ever passed the English language arts section 
language arts (Year 3 or 4) of the exam by the third or fourth year of high school (as cited in Rickles et al., 2013). This outcome is only 
reported as a supplemental finding. 
CAHSEE: Passed English language Passing the CAHSEE on the first attempt is a binary measure of whether a student passed the English language 
arts on first attempt arts section of the exam on the first attempt during the second year of high school (as cited in Rickles et al., 


2013). This outcome is only reported as a supplemental finding. 
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Appendix C.1: Findings included in the rating for the mathematics achievement domain 


Mean 
(ENCE MC ri) WWC calculations 
Study Sample Intervention Comparison Mean Effect Improvement 
Outcome measure sample size group group difference size index 
Herman et al., 2012 
CST: Algebra | (Year 1) Cohorts 1, 11 schools/ 277.86 265.42 12.44 0.30 +12 <.01 
2, and 3 1,718 (48.59) (32.93) 
students 
CST: Geometry (Year 1) Cohorts 1, 11 schools/ 286.84 272.98 14.26 0.37 +14 <.01 
2,and3 385 students (43.46) (35.60) 
CAHSEE: Math Scale Score Cohorts 11 schools/ 361.49 354.84 6.65 0.21 +8 < .01 
(Year 2) tand2 927 students (33.13) (28.79) 
CST: Algebra Il (Year 2) Cohorts 11 schools/ 270.21 254.16 16.05 0.35 +14 <.01 
jand2 295 students (50.71) (41.18) 
CST: Geometry (Year 2) Cohorts 11 schools/ 255.78 250.09 5.69 0.19 +7 03 
Jand2 559 students (31.54) (29.31) 
CST: Algebra Il (Year 3) Cohorts 11 schools/ 2OSMa 244.65 9.08 0.28 +11 01 
jand2 406 students (34.09) (30.35) 
CST: Summative Math Cohorts 11 schools/ PA) 243.95 27.60 0.58 +22 <.01 
(Year 3) jand2 189 students (62.23) (42.07) 
Domain average for mathematics achievement (Herman et al., 2012) 0.33 +13 Statistically 
significant 
Domain average for mathematics achievement across all studies 0.33 +13 na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are 
given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in 

an average individual’s percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average rounded to two 
decimal places; the average improvement index is calculated from the average effect size. The statistical significance of the study’s domain average was determined by the WWC. 
Some statistics may not sum as expected due to rounding. na = not applicable. CST = California Standards Test. CAHSEE = California High School Exit Exam. 


@ For Herman et al. (2012), a correction for multiple comparisons was needed but did not affect whether any of the contrasts were found to be statistically significant. The WWC 
calculated adjusted intervention group means by adding the impact of the intervention (the regression coefficient reported in the study) to the unadjusted comparison group posttest 
means. The findings presented here are based on aggregated samples across cohorts separately reported in the original study. The authors reported p-values for some results, but 
not for the aggregated analysis. The WWC calculated the p-values reported in the table. When aggregating cohorts, the WWC calculated weighted averages of the adjusted interven- 
tion means and unadjusted comparison group means. The authors provided unadjusted outcome standard deviations at the WWC’s request. Please see the WWC Procedures and 
Standards Handbook (version 3.0) p. 23 for more information. This study is characterized as having a statistically significant positive effect because at least one measure is positive 
and statistically significant, and no effects are negative and statistically significant, accounting for multiple comparisons. For more information, please refer to the WWC Procedures 
and Standards Handbook (version 3.0), p. 26. 
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Appendix C.2: Findings included in the rating for the student progression domain 


Mean 
(standard deviation) WWC calculations 
Intervention Comparison Mean Effect Improvement 

Outcome measure group group difference size index p-value 
Herman et al., 2012 
% Graduated: Year 1 Cohort2 11 schools/ 54 40 14 0.28 +11 anil 
entry sample 1,024 (na) (na) 

students 
% Graduated with A-G: Cohort 2 11 schools/ 25 16 9 0.22 +9 <.01 
Year 1 entry sample 1,024 (na) (na) 

students 
Domain average for mathematics achievement (Herman et al., 2012) 0.25 +10 Statistically 

significant 

Domain average for student progression across all studies 25) +10 na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are 
given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in 

an average individual’s percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average rounded to two 
decimal places; the average improvement index is calculated from the average effect size. The statistical significance of the study’s domain average was determined by the WWC. 
Some statistics may not sum as expected due to rounding. na = not applicable. 


@ For Herman et al. (2012), a correction for multiple comparisons was needed but did not affect whether any of the contrasts were found to be statistically significant. The WWC 
calculated adjusted intervention group means by adding the impact of the intervention (the regression coefficient reported in the study) to the unadjusted comparison group posttest 
means. The p-values presented here were reported in the original study. The authors provided unadjusted outcome standard deviations at the WWC’s request. Please see the WWC 
Procedures and Standards Handbook (version 3.0) p. 23 for more information. This study is characterized as having a statistically significant positive effect because at least one 
measure is positive and statistically significant, and no effects are negative and statistically significant, accounting for multiple comparisons. For more information, please refer to the 
WWC Procedures and Standards Handbook (version 3.0), p. 26. 
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Appendix C.3: Findings included in the rating for the school attendance domain 


Mean 
(standard deviation) WWC calculations 
Study Sample Intervention Comparison Mean Effect Improvement 
Outcome measure sample size group group difference size index p-value 
Herman et al., 2012 
School attendance (Year 1) Cohort3 11 schools/ 0.94 0.94 0.00 0.00 0 89 
916 stusdent (0.09) (0.07) 
School attendance (Year 4) Cohorts 11 schools/ 0.94 0.92 0.02 0.27 +10 <.01 
tand2 662 students (0.07) (0.08) 
Domain average for school attendance (Herman et al., 2012) 0.13 +5 Statistically 
significant 
Domain average for school attendance across all studies 0.13 +5 na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are 
given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in 

an average individual’s percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average rounded to two 
decimal places; the average improvement index is calculated from the average effect size. The statistical significance of the study’s domain average was determined by the WWC. 
Some statistics may not sum as expected due to rounding. na = not applicable. 


@ For Herman et al. (2012), the WWC calculated adjusted intervention group means by adding the impact of the intervention (the regression coefficient from the authors’ adjusted 
model) to the unadjusted comparison group posttest means. The WWC calculated adjusted intervention group means by adding the impact of the intervention (the regression coef- 
ficient reported in the study) to the unadjusted comparison group posttest means. The findings presented here are based on aggregated samples across cohorts separately reported 
in the original study. The authors reported p-values for some results, but not for the aggregated analysis. The WWC calculated the p-values reported for the aggregated samples in the 
table. When aggregating cohorts, the WWC calculated weighted averages of the adjusted intervention means and unadjusted comparison group means. The authors provided unad- 
justed outcome standard deviations at the WWC’s request. Please see the WWC Procedures and Standards Handbook (version 3.0) p. 23 for more information. This study is character- 
ized as having a statistically significant positive effect because at least one measure is positive and statistically significant, and no effects are negative and statistically significant, 
accounting for multiple comparisons. For more information, please refer to the WWC Procedures and Standards Handbook (version 3.0), p. 26. 


Green Dot Public Schools January 2018 Page 15 


WWC Intervention Report 


Appendix C.4: Findings included in the rating for the English language arts achievement domain 


Mean 
(standard deviation) WWC calculations 
Study Sample Intervention Comparison Mean Effect Improvement 
Outcome measure sample size group group difference size index p-value 
Herman et al., 2012 
CST: ELA (Year 1) Cohort3 11 schools/ 312.00 303.55 8.45 0.18 +7 <.01 
930 students (48.14) (45.46) 
CAHSEE: ELA Scale Score Cohorts 11 schools/ 355.24 353.46 1.74 0.06 +2 38 
(Year 2) tand2 935 students (30.74) (29.95) 
CST: ELA (Year 3) Cohorts 11 schools/ 299.98 297.54 2.44 0.05 +2 Sil 
jand2 752 students (51.96) (50.05) 
Domain average for English language arts achievement (Herman et al., 2012) 0.10 +4 Statistically 
significant 
Domain average for English language arts achievement across all studies 0.10 +4 na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who 
are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change 
in an average individual’s percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average rounded to 
two decimal places; the average improvement index is calculated from the average effect size. The statistical significance of the study’s domain average was determined by the 
WWC. Some statistics may not sum as expected due to rounding. na = not applicable. CST = California Standards Test. CAHSEE = California High School Exit Exam. ELA = English 
language arts. 


@ For Herman et al. (2012), a correction for multiple comparisons was needed but did not affect whether any of the contrasts were found to be statistically significant. The WWC 
calculated adjusted intervention group means by adding the impact of the intervention (the regression coefficient reported in the study) to the unadjusted comparison group posttest 
means. The findings presented here are based on aggregated samples across cohorts separately reported in the original study. The authors reported p-values for some results, but not 
for the aggregated analysis. The WWC calculated the p-values reported for the aggregated samples in the table. When aggregating cohorts, the WWC calculated weighted averages 

of the adjusted intervention means and unadjusted comparison group means. The authors provided unadjusted outcome standard deviations at the WWC’s request. Please see the 
WWC Procedures and Standards Handbook (version 3.0) p. 23 for more information. This study is characterized as having a statistically significant positive effect because at least one 
measure is positive and statistically significant, and no effects are negative and statistically significant, accounting for multiple comparisons. For more information, please refer to the 
WWC Procedures and Standards Handbook (version 3.0), p. 26. 
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Appendix D.1: Description of supplemental findings for the mathematics achievement domain 


Mean 
(standard deviation) WWC calculations 
Study Sample Intervention Comparison Mean Effect Improvement 
Outcome measure sample size group group difference size index p-value 
Herman et al., 2012? 
CST: Algebra | (Year 1) Cohort 1 5 schools/ 278.82 264.12 14.70 0.34 +13 < {Dil 
278 students (51.53) (32.24) 
CST: Algebra | (Year 1) Cohort2 11 schools/ 266.70 267.00 -0.30 -0.01 0 .90 
714 students (35.00) (35.00) 
CST: Algebra | (Year 1) Cohort 2 11 schools/ 266.88 266.11 0.77 0.02 +1 14 
alternate 795 students (36.04) (34.05) 
CST. Algebra | (Year 1) Cohort3 —- 11 schools/ 288.09 264.35 23.74 0.52 +20 <2 0il 
726 students (55.12) (31.77) 
CST: Geometry (Year 1) Cohort 1 5 schools/ 300.94 273.83 2711 0.57 +21 01 
47 students (44.45) (49.88) 
CST: Geometry (Year 1) Cohort3 —- 11 schools/ 280.66 214.32 6.34 0.16 +6 16 
179 students (47.98) (33.69) 
CAHSEE: Math Scale Score Cohort 1 5 schools/ 358.62 357.56 1.06 0.03 +1 all's) 
(Year 2) 228 students (33.34) (28.04) 
CAHSEE: Math Scale Score Cohort 2 11 schools/ 362.47 354.00 8.47 0.27 +11 < {0 
(Year 2) 699 students (33.00) (29.00) 
CAHSEE: Math Scale Score Cohort 2 11 schools/ 361.73 354,74 6.99 0.22 +9 <.01 
(Year 2) alternate 719 students (32.86) (29.21) 
CAHSEE: % Passed math on Cohort 1 5 schools/ 57 59 -2 —0.04 —2 70 
first attempt (Year 2) 228 students (na) (na) 
CAHSEE: % Passed math on Cohort 2 11 schools/ 59 51 8 0.15 +6 02 
first attempt (Year 2) 724 students (na) (na) 
CAHSEE: % Passed math on Cohort 2 11 schools/ 61 Bo 6 0.12 +5 .06 
first attempt (Year 2) alternate 726 students (na) (na) 
CAHSEE: % Ever passed Cohort 1 5 schools/ 84 70 14 0.33 +13 02 
math (Year 3) 187 students (na) (na) 
CAHSEE: % Ever passed Cohort 2 11 schools/ 82 ie 9 0.22 +9 <.01 
math (Year 3) alternate 616 students (na) (na) 
CST: Algebra Il (Year 2) Cohort 2 11 schools/ 267.99 257.00 10.99 0.24 +10 02 
229 students (48.00) (42.00) 
CST: Algebra Il (Year 2) Cohort2 11 schools/ 268.16 256.79 Wai 0.25 +10 02 
alternate 247 students (47.34) (41.49) 
CST: Geometry (Year 2) Cohort 1 5 schools/ 297.38 244.31 13.07 0.45 +17 01 
151 students (32.98) (22.69) 
CST: Geometry (Year 2) Cohort 2 11 schools/ 255.12 252.00 SZ 0.10 +4 26 
408 students (31.00) (31.00) 
CST: Geometry (Year 2) Cohort 2 11 schools/ 200123 251.93 3.30 0.11 +4 22 
alternate 446 students (30.51) (29.51) 
CST. Algebra Il (Year 3) Cohort 1 5 schools/ 247.30 243.64 3.66 0.14 +5 48 
106 students (27.47) (25.38) 
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Mean 
(standard deviation) WWC calculations 
Study Sample Intervention Comparison Mean Effect Improvement 

Outcome measure sample size group group difference size index p-value 
CST: Algebra Il (Year 3) Cohort2 11 schools/ 255.34 242.90 12.44 0.36 +14 <.01 

alternate 316 students (35.89) (31.83) 
CST: Summative Math Cohort 1 5 schools/ 273.67 243.83 29.84 0.50 +19 04 
(Year 3) 50 students (63.43) (52.60) 
CST: Summative Math Cohort 2 11 schools/ 270.80 244.00 26.80 0.61 +23 < .01 
(Year 3) 139 students (48.00) (38.00) 
CST: Summative Math Cohort 2 11 schools/ 269.89 247.65 22.24 0.49 +19 < .01 
(Year 3) alternate 151 students (46.77) (43.38) 
CAHSEE: % Ever passed Cohort 1 5 schools/ 90 92 -2 -0.07 -3 A4 
math (Year 4) 166 students (na) (na) 
CAHSEE: % Ever passed Cohort 2 11 schools/ 96 91 5 0.20 +8 .03 
math (Year 4) 496 students (na) (na) 
CAHSEE: % Ever passed Cohort 2 11 schools/ 69 61 8 0.17 +7 01 
math (Year 1 entry sample) 1,024 (na) (na) 

students 


Table Notes: The supplemental findings presented in this table are additional findings from studies in this report that meet WWC design standards with or without reservations, 
but do not factor into the determination of the intervention rating. For mean difference, effect size, and improvement index values reported in the table, a positive number favors 
the intervention group and a negative number favors the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing 
the average change expected for all individuals who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate 
presentation of the effect size, reflecting the change in an average individual’s percentile rank that can be expected if the individual is given the intervention. Some statistics may 
not sum as expected due to rounding. na = not applicable. CST = California Standards Test. CAHSEE = California High School Exit Exam. 


4 For Herman et al. (2012), a correction for multiple comparisons was needed and resulted in a change in the statistical significance for one finding: the author-reported p-value for 
CST: Summative Math (Year 3) for Cohort 1 was .041, and the WWC-computed critical p-value was .032. Therefore, the WWC does not find the result for this finding to be statistically 
significant. The p-values presented here were reported in the original study. The WWC calculated adjusted intervention group means by adding the impact of the intervention (the 
regression coefficient from the authors’ adjusted model) to the unadjusted comparison group posttest means. The authors provided unadjusted outcome standard deviations at the 
WWC’s request. Please see the WWC Procedures and Standards Handbook (version 3.0) p. 23 for more information. 
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Appendix D.2: Description of supplemental findings for the school progression domain 


Mean 
(standard deviation) WWC calculations 
Study Sample Intervention Comparison Mean Effect Improvement 

Outcome measure sample size group group difference size index p-value 
Herman et al., 2012 
% Graduated: Year 4 Sample —_ Cohort 1 5 schools/ 79 55 24 0.52 +20 <n (Oil 

166 students (na) (na) 
% Graduated: Year 4 Sample —_ Cohort 2 11 schools/ 88 70 18 0.44 +17 < .01 

496 students (na) (na) 
% Graduated w/A-G: Cohort 1 5 schools/ 47 13 34 0.78 +28 < .01 
Year 4 Sample 166 students (na) (na) 
% Graduated w/A-G: Cohort 2 11 schools/ 44 27 il 0.36 +14 < .01 
Year 4 Sample 496 students (na) (na) 


Table Notes: The supplemental findings presented in this table are additional findings from studies in this report that meet WWC design standards with or without reservations, 
but do not factor into the determination of the intervention rating. For mean difference, effect size, and improvement index values reported in the table, a positive number favors 
the intervention group and a negative number favors the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing 
the average change expected for all individuals who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate 
presentation of the effect size, reflecting the change in an average individual’s percentile rank that can be expected if the individual is given the intervention. Some statistics may 
not sum as expected due to rounding. na = not applicable. 


@ For Herman et al. (2012), a correction for multiple comparisons was needed but did not affect whether any of the contrasts were found to be statistically significant. The p-values 
presented here were reported in the original study. The WWC calculated adjusted intervention group means by adding the impact of the intervention (the regression coefficient from 
the authors’ adjusted model) to the unadjusted comparison group posttest. The authors provided unadjusted outcome standard deviations at the WWC’s request. Please see the WWC 
Procedures and Standards Handbook (version 3.0) p. 23 for more information. 
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Appendix D.3: Description of supplemental findings for the school attendance domain 


GET 
(standard deviation) WWC calculations 
Study Sample Intervention Comparison Mean Effect Improvement 
Outcome measure sample size group group difference size index p-value 
Herman et al., 2012 
School attendance: Year 1 Cohort 1 5 schools/ 0.93 0.93 0.00 0.00 0 .99 
330 students (0.10) (0.07) 
School attendance: Year 1 Cohort 2 11 schools/ 0.93 0.93 0.00 0.00 0 .66 
888 students (0.08) (0.08) 
School attendance: Year 1 Cohort 2 11 schools/ 0.92 0.92 0.00 0.00 0 90 
alternate 978 students (0.08) (0.08) 
School attendance: Year 2 Cohort 1 5 schools/ 0.93 0.94 -0.01 -0.10 -4 43 
242 students (0.13) (0.05) 
School attendance: Year 2 Cohort 2 11 schools/ 0.93 0.93 0.00 na na 87 
718 students (nr) (nr) 
School attendance: Year 2 Cohort 2 11 schools/ 0.93 0.93 0.00 0.00 0 54 
alternate 779 students (0.11) (0.08) 
School attendance: Year 3 Cohort 1 5 schools/ 0.95 0.93 0.02 0.31 +12 04 
187 students (0.06) (0.07) 
School attendance: Year 3 Cohort 2 11 schools/ 0.94 0.94 0.00 0.00 0 94 
564 students (0.07) (0.07) 
School attendance: Year 3 Cohort 2 11 schools/ 0.93 0.93 0.00 0.00 0 48 
alternate 622 students (0.08) (0.08) 
School attendance: Year 4 Cohort 1 5 schools/ 0.94 0.91 0.03 0.35 +14 04 
166 students (0.08) (0.09) 
School attendance: Year 4 Cohort 2 11 schools/ 0.94 0.93 0.01 0.13 +5 02 
496 students (0.07) (0.08) 


Table Notes: The supplemental findings presented in this table are additional findings from studies in this report that meet WWC design standards with or without reservations, 
but do not factor into the determination of the intervention rating. For mean difference, effect size, and improvement index values reported in the table, a positive number favors 
the intervention group and a negative number favors the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing 
the average change expected for all individuals who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate 
presentation of the effect size, reflecting the change in an average individual’s percentile rank that can be expected if the individual is given the intervention. Some statistics may 
not sum as expected due to rounding. nr = not reported. 


@ For Herman et al. (2012), a correction for multiple comparisons was needed and resulted in a change in the statistical significance for three findings: the author-reported p-value 
for Student attendance: Year 3 for Cohort 1 was .043, and the WWC-computed critical p-value was .014; the author-reported p-value for Student attendance: Year 4 for Cohort 1 was 
.036, and the WWC-computed critical p-value was .009; the author-reported p-value for Student attendance: Year 4 for Cohort 2 was .024, and the WWC-computed critical p-value 
was .005. Therefore, the WWC does not find the result for any of these three findings to be statistically significant. The WWC calculated adjusted intervention group means by adding 
the impact of the intervention (the regression coefficient from the authors’ adjusted model) to the unadjusted comparison group posttest. The authors provided unadjusted outcome 
standard deviations at the WWC’s request. Please see the WWC Procedures and Standards Handbook (version 3.0) p. 23 for more information. The authors did not provide standard 
deviations for the Student attendance: Year 2 outcome for Cohort 2, so the WWC cannot calculate an effect size. 
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Appendix D.4: Description of supplemental findings for the English language arts achievement domain 


Mean 
(standard deviation) WWC calculations 
Study Sample Intervention Comparison Mean Effect Improvement 
Outcome measure sample size group group difference size index p-value 
Herman et al., 2012 
CST: ELA (Year 1) Cohort 1 5 schools/ 314.72 305.50 9.22 0.21 +8 <.01 
330 students (41.50) (47.40) 
CST: ELA (Year 1) Cohort2 11 schools/ 304.42 301.00 3.42 0.08 +3 07 
888 students (44.00) (44.00) 
CST: ELA (Year 1) Cohort2 11 schools/ 304.50 301.57 2.93 0.07 +3 10 
alternate 978 students (43.92) (43.74) 
CAHSEE: ELA Scale Score Cohort 1 5 schools/ 353.00 354,94 -1.94 -0.07 -3 A7 
(Year 2) 230 students (29.77) (29.86) 
CAHSEE: ELA Scale Score Cohort 2 11 schools/ 355,94 353.00 2.94 0.10 +4 .06 
(Year 2) 705 students (31.00) (30.00) 
CAHSEE: ELA Scale Score Cohort 2 11 schools/ Boos 353.74 3.79 0.12 +5 01 
(Year 2) alternate 725 students (30.23) (31.01) 
CAHSEE: % Passed ELA on Cohort 1 5 schools/ 54 62 -8 -0.16 —6 16 
first attempt (Year 2) 230 students (na) (na) 
CAHSEE: % Passed ELA on Cohort 2 11 schools/ 59 51 8 0.15 +6 01 
first attempt (Year 2) 724 students (na) (na) 
CAHSEE: % Passed ELA on Cohort 2 11 schools/ 60 54 6 0.12 +5 04 
first attempt (Year 2) alternate 725 students (na) (na) 
CST: ELA (Year 2) Cohort 1 5 schools/ 303.29 300.01 3.28 0.07 +3 A4 
242 students (46.74) (47.25) 
CST: ELA (Year 2) Cohort2 11 schools/ 299.82 296.00 3.82 0.09 +3 10 
724 students (47.00) (43.00) 
CST: ELA (Year 2) Cohort2 11 schools/ 299.13 295.14 3.99 0.09 +4 07 
alternate 786 students (46.76) (43.80) 
CST: ELA (Year 3) Cohort 1 5 schools/ 291.77 296.15 —4,38 —0.09 -3 A4 
188 students (51.35) (50.43) 
CST: ELA (Year 3) Cohort 2 11 schools/ 302.71 298.00 47] 0.09 +4 li 
564 students (52.00) (50.00) 
CST: ELA (Year 3) Cohort 2 11 schools/ 301.52 298.48 3.04 0.06 +2 A0 
alternate 622 students (52.06) (49.44) 
CAHSEE: % Ever passed ELA — Cohort 1 5 schools/ 76 us 3 0.07 +3 59 
(Year 3) 187 students (na) (na) 
CAHSEE: % Ever passed ELA Cohort2 _—_‘_11 schools/ 79 us 6 0.14 +6 01 
(Year 3)° alternate 616 students (na) (na) 
CAHSEE: % Ever passed ELA Cohort 1 5 schools/ 86 90 —-4 -0.12 —5 35 
(Year 4) 166 students (na) (na) 
CAHSEE: % Ever passed ELA Cohort2 ‘11 schools/ 94 90 4 0.14 +6 .07 
(Year 4) 496 students (na) (na) 
CAHSEE: % Ever passed ELA — Cohort 2 11 schools/ 69 61 8 0.17 +7 < .01 
(Year 1 entry sample) 1,024 (na) (na) 
students 
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Table Notes: The supplemental findings presented in this table are additional findings from studies in this report that meet WWC design standards with or without reservations, 
but do not factor into the determination of the intervention rating. For mean difference, effect size, and improvement index values reported in the table, a positive number favors 
the intervention group and a negative number favors the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing 
the average change expected for all individuals who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate 
presentation of the effect size, reflecting the change in an average individual’s percentile rank that can be expected if the individual is given the intervention. Some statistics may 
not sum as expected due to rounding. na = not applicable. CST = California Standards Test. CAHSEE = California High School Exit Exam. ELA = English language arts. 


@ For Herman et al. (2012), a correction for multiple comparisons was needed and resulted in a change in the statistical significance for four findings: the author-reported p-value for 
CAHSEE: ELA scale score (Year 2) for the alternate Cohort 2 outcome was .014, and the WWC-computed critical p-value was .013; the author-reported p-value for the CAHSEE: % 
Passed ELA on the first attempt (Year 2) for Cohort 2 outcome was .012, and the WWC-computed critical p-value was .010; the author-reported p-value for the CAHSEE: % Passed 
ELA on the first attempt (Year 2) for the alternate Cohort 2 outcome was .035, and the WWC-computed critical p-value was .015; the author-reported p-value for the CAHSEE: % Ever 
Passed ELA (Year 3) for the alternate Cohort 2 outcome was .008, and the WWC-computed critical p-value was .0075. Therefore, the WWC does not find the result for any of these four 
outcomes to be statistically significant. The p-values presented here were reported in the original study. The WWC calculated adjusted intervention group means by adding the impact 
of the intervention (the regression coefficient from the authors’ adjusted model) to the unadjusted comparison group posttest. The authors provided unadjusted outcome standard 
deviations at the WWC’s request. Please see the WWC Procedures and Standards Handbook (version 3.0) p. 23 for more information. 


> In response to an author query for Herman et al. (2012), the authors provided a corrected sample size different from what was originally published. The study originally reported the 
intervention sample size was 308 for this outcome, but updated the sample to 309 in their response to the query. The updated number is included here. 
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Endnotes 


' The descriptive information for this intervention comes from publicly available sources: the intervention website (Green Dot Public 
Schools website at http://greendot.org/about/, downloaded in April 2017) and Dillon, S. (2010, June). School is turned around, but 
cost gives pause. Retrieved from http://www.nytimes.com/). The What Works Clearinghouse (WWC) requests developers review the 
intervention description sections for accuracy from their perspective. The WWC provided the developer with the intervention descrip- 
tion in April 2017; however, the WWC did not receive a response. Further verification of the accuracy of the descriptive information for 
this intervention is beyond the scope of this review. 


? The literature search reflects documents publicly available by February 2017. Reviews of the studies in this report used the standards 
from the WWC Procedures and Standards Handbook (version 3.0) and the Charter Schools review protocol (version 3.0). The evidence 
presented in this report is based on available research. Findings and conclusions could change as new research becomes available. 


3 Please see the Charter Schools review protocol (version 3.0) for a list of all outcome domains. 


4 For criteria used to determine the rating of effectiveness and extent of evidence, see the WWC Rating Criteria on p. 24. These 
improvement index numbers show the average and range of individual-level improvement indices for all findings across the studies. 


5 The sample sizes reported here represent the largest non-overlapping samples of students from Cohorts 1, 2, and 3 in each domain 
across findings that contribute to the intervention rating (i.e., those reported in Appendix C). 


6 Please see the Charter Schools review protocol (version 3.0) for details on the types of interventions that are eligible for review. A 
study of the effectiveness of an individual charter school is eligible to be included in a review of the evidence of the effectiveness of an 
individual charter school, but is not eligible to be included in a review of the evidence of the effectiveness of anamed CMO or charter 
network, like Green Dot Public Schools. 

’ The findings from Cohort 2 that contribute to the effectiveness rating are based on a sample with 4 years of outcomes available 
(results presented in Rickles et al., 2013). Findings from Cohort 2 based on an alternate sample with only 3 years of outcomes (results 
presented in Herman et al., 2012) are reported as supplemental findings, which do not contribute to the effectiveness rating in this 
intervention report. These alternate findings for Cohort 2 are reported in Appendix D. 

8 The 10 student outcome domains are: mathematics achievement, English language arts achievement, science achievement, social 
studies achievement, general achievement, social-emotional competence, disciplinary experiences, student attendance, student pro- 
gression, and earnings in adulthood. 

° Students take mathematics CST exams in different subjects each year. For example, students who take the Algebra | exam in Year 1 
generally take the Geometry exam in Year 2, while other students take Geometry in Year 1 and Algebra II in Year 2. Because the set of 
students that take each exam differs by year, the subject-specific tests contribute to the effectiveness rating in each year they are given. 


Recommended Citation 


What Works Clearinghouse, Institute of Education Sciences, U.S. Department of Education. (2018, January). 
Charter Schools intervention report: Green Dot Public Schools. Retrieved from http://whatworks.ed.gov 
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WWC Rating Criteria 
Criteria used to determine the rating of a study 


Study rating Criteria 

Meets WWC group design A study that provides strong evidence for an intervention’s effectiveness, such as a well-implemented RCT. 
standards without reservations 

Meets WWC group design A study that provides weaker evidence for an intervention’s effectiveness, such as a QED or an RCT with high 
standards with reservations attrition that has established equivalence of the analytic samples. 


Criteria used to determine the rating of effectiveness for an intervention 


Rating of effectiveness Criteria 


Positive effects Two or more studies show statistically significant positive effects, at least one of which met WWC group design 
standards without reservations, AND 
No studies show statistically significant or substantively important negative effects. 


Potentially positive effects At least one study shows a statistically significant or substantively important positive effect, AND 
No studies show a statistically significant or substantively important negative effect AND fewer or the same number 
of studies show indeterminate effects than show statistically significant or substantively important positive effects. 


Mixed effects At least one study shows a statistically significant or substantively important positive effect AND at least one study 
shows a statistically significant or substantively important negative effect, but no more such studies than the number 
showing a statistically significant or substantively important positive effect, OR 
At least one study shows a statistically significant or substantively important effect AND more studies show an 
indeterminate effect than show a statistically significant or substantively important effect. 


Potentially negative effects One study shows a statistically significant or substantively important negative effect and no studies show a statisti- 
cally significant or substantively important positive effect, OR 
Two or more studies show statistically significant or substantively important negative effects, at least one study 
shows a statistically significant or substantively important positive effect, and more studies show statistically 
significant or substantively important negative effects than show statistically significant or substantively important 
positive effects. 


Negative effects Two or more studies show statistically significant negative effects, at least one of which met WWC group design 
standards without reservations, AND 
No studies show statistically significant or substantively important positive effects. 


No discernible effects None of the studies shows a statistically significant or substantively important effect, either positive or negative. 


Criteria used to determine the extent of evidence for an intervention 


Extent of evidence Criteria 


Medium to large The domain includes more than one study, AND 
The domain includes more than one school, AND 
The domain findings are based on a total sample size of at least 350 students, OR, assuming 25 students in a class, 
a total of at least 14 classrooms across studies. 


Small The domain includes only one study, OR 
The domain includes only one school, OR 
The domain findings are based on a total sample size of fewer than 350 students, AND, assuming 25 students in a 
Class, a total of fewer than 14 classrooms across studies. 
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Attrition 


Baseline 


Clustering Adjustment 


Confounding factor 


Design 


Effect size 
Eligibility 


Equivalence 


Attrition occurs when an outcome variable is not available for all subjects initially assigned 
to the intervention and comparison groups. If a randomized controlled trial (RCT) or regres- 
sion discontinuity design (RDD) study has high levels of attrition, the validity of the study 
results can be called into question. An RCT with high attrition cannot receive the highest 
rating of Meets WWC Group Design Standards without Reservations, but can receive a 
rating of Meets WWC Group Design Standards with Reservations if it establishes baseline 
equivalence of the analytic sample. Similarly, the highest rating an RDD with high attrition 
can receive is Meets WWC RDD Standards with Reservations. 


For single-case design research, attrition occurs when an individual fails to complete all 
required phases or data points in an experiment, or when the case is a group and indi- 
viduals leave the group. If a single-case design does not meet minimum requirements for 
phases and data points within phases, the study cannot receive the highest rating of Meets 
WWC Pilot Single-Case Design Standards without Reservations. 


A point in time before the intervention was implemented in group design research and in 
regression discontinuity design studies. When a study is required to satisfy the baseline 
equivalence requirement, it must be done with characteristics of the analytic sample at 
baseline. In a single-case design experiment, the baseline condition is a period during 
which participants are not receiving the intervention. 


An adjustment to the statistical significance of a finding when the units of assignment 

and analysis differ. When random assignment is carried out at the cluster level, outcomes 
for individual units within the same clusters may be correlated. When the analysis is con- 
ducted at the individual level rather than the cluster level, there is a mismatch between 
the unit of assignment and the unit of analysis, and this correlation must be accounted for 
when assessing the statistical significance of an impact estimate. If the correlation is not 
accounted for in a mismatched analysis, the study may be too likely to report statistically 
significant findings. To fairly assess an intervention’s effects, in cases where study authors 
have not corrected for the clustering, the WWC applies an adjustment for clustering when 
reporting statistical significance. 


A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 


The method by which intervention and comparison groups are assigned (group design and 
regression discontinuity design) or the method by which an outcome measure is assessed 
repeatedly within and across different phases that are defined by the presence or absence 
of an intervention (single-case design). Designs eligible for WWC review are randomized 
controlled trials, quasi-experimental designs, regression discontinuity designs, and single- 
case designs. 


The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 


A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 


A demonstration that the analytic sample groups are similar on observed characteristics 
defined in the review area protocol. 
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Extent of evidence 


Gain scores 


Group design 


Improvement index 


Intervention 


Intervention report 


Multiple comparison 
adjustment 


An indication of how much evidence from group design studies supports the findings in an 
intervention report. The extent of evidence categorization for intervention reports focuses 
on the number and sizes of studies of the intervention in order to give an indication of how 
broadly findings may be applied to different settings. There are two extent of evidence cat- 
egories: small and medium to large. 


¢ small: includes only one study, or one school, or findings based on a total sample size of 
less than 350 students and 14 classrooms (assuming 25 students in a class) 


e¢ medium to large: includes more than one study, more than one school, and findings 
based on a total sample of at least 350 students or 14 classrooms 


The result of subtracting the pretest from the posttest for each individual in the sample. 
Some studies analyze gain scores instead of the unadjusted outcome measure as a method 
of accounting for the baseline measure when estimating the effect of an intervention. The 
WWC reviews and reports findings from analyses of gain scores, but gain scores do not 
satisfy the WWC’s requirement for a statistical adjustment under the baseline equivalence 
requirement. This means that a study that must satisfy the baseline equivalence require- 
ment and has baseline differences between 0.05 and 0.25 standard deviations Does Not 
Meet WWC Group Design Standards if the study’s only adjustment for the baseline measure 
was in the construction of the gain score. 


A study design in which outcomes for a group receiving an intervention are compared to 
those for a group not receiving the intervention. Comparison group designs eligible for 
WWC review are randomized controlled trials and quasi-experimental designs. 


Along a percentile distribution of individuals, the improvement index represents the gain or 
loss of the average individual due to the intervention. As the average individual starts at the 
50th percentile, the measure ranges from —50 to +50. 


An educational program, product, practice, or policy aimed at improving student outcomes. 


A summary of the findings of the highest-quality research on a given program, product, 
practice, or policy in education. The WWC searches for all research studies on an interven- 
tion, reviews each against design standards, and summarizes the findings of those that 
meet WWC design standards. 


An adjustment to the statistical significance of results to account for multiple comparisons 
in a group design study. The WWC uses the Benjamini-Hochberg (BH) correction to adjust 
the statistical significance of results within an outcome domain when study authors perform 
multiple hypothesis tests without adjusting the p-value. The BH correction is used in three 
types of situations: studies that tested multiple outcome measures in the same outcome 
domain with a single comparison group; studies that tested a given outcome measure 
with multiple comparison groups; and studies that tested multiple outcome measures in 
the same outcome domain with multiple comparison groups. Because repeated tests of 
highly correlated constructs will lead to a greater likelihood of mistakenly concluding that 
the impact was different from zero, in all three situations, the WWC uses the BH correction 
to reduce the possibility of making this error. The WWC makes separate adjustments for 
primary and secondary findings. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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Outcome domain 
Quasi-experimental 
design (QED) 


Randomized controlled 
trial (RCT) 


Rating of effectiveness 


Regression 
discontinuity design 
(RDD) 


Single-case design 


Standard deviation 


Statistical significance 


Study rating 


Substantively important 


Systematic review 


A group of closely-related outcomes. A domain is the organizing construct for a set of 
related outcomes through which studies claim effectiveness. 


A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 


A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 


For group design research, the WWC rates the effectiveness of an intervention in each 
domain based on the quality of the research design and the magnitude, statistical signifi- 
cance, and consistency in findings. For single-case design research, the WWC rates the 
effectiveness of an intervention in each domain based on the quality of the research design 
and the consistency of demonstrated effects. The criteria for the ratings of effectiveness are 
given in the WWC Rating Criteria on p. 24. 


A design in which groups are created using a continuous scoring rule. For example, stu- 
dents may be assigned to a summer school program if they score below a preset point on a 
standardized test, or schools may be awarded a grant based on their score on an applica- 
tion. A regression line or curve is estimated for the intervention group and similarly for the 
comparison group, and an effect occurs if there is a discontinuity in the two regression lines 
at the cutoff. 


A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 


The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 


Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statisti- 
cally significant if the likelihood that the difference is due to chance is less than 5% (p < .05). 


The result of the WWC assessment of a study. The rating is based on the strength of the 
evidence of the effectiveness of the educational intervention. Studies are given a rating of 
Meets WWC Design Standards without Reservations, Meets WWC Design Standards with 
Reservations, or Does Not Meet WWC Design Standards, based on the assessment of the 
study against the appropriate design standards. The WWC has design standards for group 
design, single-case design, and regression discontinuity design studies. 


A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


A review of existing literature on a topic that is identified and reviewed using explicit meth- 
ods. AWWC systematic review has five steps: 1) developing a review protocol; 2) searching 
the literature; 3) reviewing studies, including screening studies for eligibility, reviewing the 
methodological quality of each study, and reporting on high quality studies and their find- 
ings; 4) combining findings within and across studies; and, 5) summarizing the review. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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An intervention report summarizes the findings of high-quality research on a given program, practice, or policy in 
education. The WWC searches for all research studies on an intervention, reviews each against evidence standards, 
and summarizes the findings of those that meet standards. 


This intervention report was prepared for the WWC by Mathematica Policy Research under contract ED-IES-13-C-0010. 
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